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1  Summary 

The  purpose  of  this  program  is  to  address  the  development  of  algorithms  for  adaptive 
processing  of  multi-sensor  data,  employing  feedback  to  optimize  the  linkage  between 
observed  data  and  sensor  control.  The  envisioned  multi-modal  adaptive  system  is 
applicable  for  intelligence,  surveillance,  and  reconnaissance  (ISR)  in  general 
environments,  addressing  base  and  port  security,  as  well  as  urban  and  suburban  sensing 
during  wartime  and  peace-keeping  operations.  Of  significant  importance  for  current  and 
anticipated  DoD  activities,  the  ISR  system  is  designed  to  detect  asymmetric  threats,  with 
the  goal  of  recognizing  unusual  behavior  or  activities.  Technologies  and  systems 
developed  under  this  effort  will  be  designed  for  semi-automated  scene  awareness,  with 
the  objective  of  recognizing  behavior  that  appears  atypical  (e.g.  atypical  object  motion, 
and  dynamic  characteristics  of  people  and  vehicles).  Leveraging  our  previously 
developed  technology,  SIG  is  developing  second-generation  methods  to  adaptively  leam 
the  statistics  of  dynamic  object  behavior  in  video,  while  focusing  on  defining  system 
requirements  for  sensor  deployment  by  using  field  data  (vs.  highly  controlled  indoor 
data).  SIG  is  also  working  closely  with  its  subcontractor,  Lockheed  Martin,  to  integrate 
additional  technologies,  such  as  object  classification  and  recognition,  to  provide  a  more 
robust  and  discriminative  system.  Additionally,  SIG  is  cooperating  with  the  Navy’s 
China  Lake  facility  to  collect  representative  data  for  a  deployed  system,  and  to  specify 
requirements  and  features  necessary  of  such  a  system.  Finally,  SIG  is  coordinating  with 
Integrian  on  prototype  development  schedules  and  product  integration  requirements,  and 
defining  a  joint  marketing  and  commercialization  strategy  for  such  products. 

2  Technical  Developments 

During  the  last  reporting  period,  important  progress  has  been  made  in  the  system  for  both 
the  object  tracking  and  anomalous  behavior  algorithms.  We  have  completed  significant 
enhancements  to  the  background  color  modeling  that  improves  both  the  efficiency  and 
performance  of  the  overall  system,  allowing  more  consistent  and  robust  object  tracking 
and  anomalous  behavior  detection.  In  addition,  we  have  completed  several  data  collection 
programs,  including  efforts  that  will  support  analysis  for  multi-camera  analysis  as  well  as 
combined  video  and  acoustic  tracking  capabilities.  SIG  has  also  completed  an  initial 
implementation  of  our  real-time  analytics  framework  that  can  support  the  testing  process. 
Further  motivation  for  developing  real-time  video  analytics  is  the  desire  to  transition  this 
technology  into  fielded  systems. 

2.1  Tracking  Algorithms 

SIG  has  continued  to  test  and  stabilize  the  tracking  algorithm.  Modeling  improvements 
have  been  made  to  the  shape  and  color  models  for  computational  efficiency  and 
robustness  across  varying  environments.  In  particular,  SIG  has  implemented  and  tested  a 
novel  approach  to  the  background  color  modeling  that  has  improved  both  the  efficiency 
and  the  performance  of  the  overall  system.  This  novel  approach  for  the  background 
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model  is  able  dynamically  track  a  changing  background  color  distribution  through  the  use 
of  multiple  hypothesis  testing  based  on  previous  background  pixel  observations. 

The  incorporation  of  this  new  background  modeling  approach  has  dramatically  improved 
the  performance  of  the  object  detection  and  tracking  systems  relative  to  other  approaches. 
This  novel  approach  is  not  present  in  the  literature  on  the  subject  and  has  resulted  in 
better  object  identification  and  tracking  due  to  the  improved  fit  of  the  background  color 
model  distribution  with  observed  background  data,  particularly  in  scenes  with  complex 
and/or  dynamic  background  features  (such  as  trees  swaying  the  wind  or  variable  lighting 
conditions).  In  addition,  the  new  multi-hypothesis  background  model  approach  has 
resulted  in  better  object  tracking  due  to  more  robust  object  identification  and  assignments 
when  there  are  color  occlusions  between  the  background  and  moving  foreground  objects. 


Figure  1:  Improved  Background  model  based  multi-hypothesis  approach  results  in  better 
fit  to  dynamic  background  color  distributions  and  provides  better  tracking  performance  in 
complex  dynamic  scenes. 

2.2  Anomalous  Behavior  Analysis 

One  of  the  key  improvements  SIG  has  introduced  into  the  system  in  the  previous 
reporting  period  the  ability  to  detect  anomalous  behavior.  Over  a  short  training  period,  an 
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anomalous  behavior  model  is  trained  for  “normal”  behavior;  that  is,  behaviors  that  are 
likely  to  be  seen  based  upon  observed  object  positions  and  speeds.  We  have  continued  to 
refine  this  approach  and  have  begun  to  analyze  models  where  the  decisions  about 
anomalous  behavior  can  be  based  on  the  total  behavior  history  of  the  tracked  object  as 
opposed  to  making  decisions  based  simply  on  estimates  of  current  velocity  and  position. 
As  these  more  complete  models  of  object  activity  are  refined,  we  will  be  able  to  make 
more  sophisticated  judgments  about  anomalous  behavior.  At  the  same  time,  the  system 
will  still  be  able  to  learn  typical  behaviors  directly  from  the  data,  and  characterize 
anomalies  as  any  deviation  from  this  expected  activity  -  without  the  need  for  periodic 
behavior  rule  updates. 

2.3  Multi-Sensor  Data  Collections  Completed 

During  this  last  reporting  period,  SIG  has  worked  with  AFRL  to  complete  a  multi-camera 
video  data  collection  effort.  The  data  collection  site  has  been  designated  at  the  Air  Force 
Research  Laboratory  facility,  Building  620  and  surrounding  vicinity  situated  on  Wright- 
Patterson  AFB,  OH.  The  data  collection  was  designed  to  support  multi-camera  video 
analysis  using  our  existing  analytics  framework.  The  data  collected  included  over  134 
hours  of  high-resolution  video  data,  collected  over  a  three-day  period.  The  data  collection 
was  designed  to  capture  pedestrian  and  vehicular  traffic  over  a  variety  of  different 
lighting  and  weather  conditions,  including  both  daytime  and  nighttime.  The  data  was 
collected  using  four  video  cameras  sited  with  partially  overlapping  fields-of-view  to 
support  analysis  of  object  tracking  across  multiple  video  streams. 

In  addition  to  the  multi-camera  data  collection,  we  have  also  collected  initial  dual-mode 
data  collection  whereby  the  SIG  video  analytics  will  be  paired  with  adaptive  acoustic 
array  data.  The  data  collected  will  be  post-processed  to  demonstrate  a  capability  to 
simultaneously  track  an  object  and  beam-form  an  acoustic  array  to  listen  to  that  object  as 
it  moves  through  the  field  of  view.  . 

2.4  Sensor  Management 

SIG  has  begun  initial  design  of  the  sensor  management  capabilities  that  will  help  achieve 
our  high  level  project  goal  of  employing  feedback  to  optimize  the  linkage  between 
observed  data  and  sensor  control.  Our  goal  is  to  implement  an  approach  that  creates  the 
framework  necessary  for  SMA  that  provides  multiple  levels  of  resolution  for  different 
processing  functions.  In  addition,  this  same  framework  will  be  useful  as  we  extend  the 
same  concept  to  additional  types  of  sensors  or  processing  regimes  that  are  controlled  by 
the  low  level  object  detection  and  tracking  engine  to  extract  even  more  information  about 
the  objects  of  interest  that  are  being  examined  in  the  process  of  asymmetric  threat 
detection. 

2.5  Real-Time  Implementation 

SIG  has  made  considerable  progress  its  real-time  implementation  efforts.  While  these  are 
not  in-scope  with  respect  to  specific  C2CS  contractual  deliverables,  this  real-time 
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capability  provides  for  accelerated  testing  and  performance  assessment  on  large 
quantities  of  data.  SIG  has  converted  all  of  the  core  object  detection  and  tracking 
algorithms  to  efficient  C  code  implementations  while  maintaining  the  appropriate 
functional  interfaces  to  SIG’s  other  Matlab-based  object  behavior  analysis  code.  SIG  is 
also  investigating  high  performance  sensor  platforms  that  can  provide  hardware  and 
DSP-based  acceleration  for  the  core  analytics  modeling  algorithms,  as  well  as  supporting 
to  capability  for  critical  modeling  tasks  to  access  sensor  data  at  full  resolution  before  any 
compression  effects  or  artifacts  are  introduced. 

3  Programmatic  Developments 

During  the  last  perfonnance  period,  SIG  hosted  a  meeting  for  the  C2CS  project  sponsors 
and  key  members  of  the  research  teams.  The  meeting  was  held  on  July  24,  2007  at  the 
offices  of  Signal  Innovations  Group  (SIG),  in  Durham,  NC  and  attendees  included  Drs. 
Behzad  Kamgar-Parsi  and  Ken  Shaw  from  ONR;  Lawrence  Garin,  Paul  Runkle  and  Matt 
Welborn  from  SIG;  Abhijit  Mahalanobis  from  Lockheed  Martin;  and  Alan  Van  Nevel 
from  NAVAIR.  This  meeting  included  summary  presentations  of  the  current  research 
from  all  teams  as  well  as  plans  for  the  remaining  contract  period.  In  addition,  this 
meeting  provided  an  occasion  to  discuss  proposals  for  the  Year  3  option  of  this  C2CS 
project  as  well  as  opportunities  for  possible  transition  of  the  technology,  so  that  the 
research  may  be  further  enhanced  and  progress  closer  to  other  DoD  critical  applications. 
One  particular  transition  opportunity  is  the  DARPA  Large  Area  Coverage  Optical 
Search- While-Track  and  Engage  (LACOSTE)  project.  That  program  is  now  completing 
Phase  I,  and  it  is  deemed  that  much  of  the  technology  developed  under  this  current  SBIR 
program  may  be  transitioned  to  that  DARPA  project.  Future  Directions 

As  already  noted,  the  key  technologies  developed  under  this  C2CS  project  are  very 
relevant  to  the  DARPA  LACOSTE  effort.  Particular  technologies  that  that  have  been 
developed  under  this  program  and  that  are  of  interest  for  LACOSTE  are:  (i)  automated 
tracking  of  multiple  targets  through  complex  scenes,  including  maintenance  of  multiple 
hypotheses  through  occlusions;  (ii)  adaptive  learning  and  removal  of  complex 
background  characteristics;  (iii)  learning  statistical  models  for  "normal"  behavior;  and 
(iv)  development  of  sensor-management  concepts  that  adaptively  exploit  the  full  sensor 
capabilities,  through  the  directed  acquisition  of  data  to  minimize  model  and  hypothesis 
uncertainty.  A  more  detailed  plan  of  possible  research  directions  for  Year  3  of  this  C2CS 
effort  has  been  prepared  and  submitted  to  ONR  in  August  2007.  These  proposals  would 
help  to  not  only  support  current  C2CS  objectives,  but  also  position  the  technology  to 
other  efforts  such  as  DARPA  LACOSTE  program. 


